Double Binds and Double Blinds: Evaluation Tactics in Critically Oriented HCI

Critically oriented researchers within Human-Computer Interaction (HCI) have fruitfully intersected design and critical analysis to engage users and designers in reflection on underlying values, assumptions and dominant practices in technology. To successfully integrate this work within the HCI community, critically oriented researchers have tactically engaged with dominant practices within HCI in the design and evaluation of their work. This paper draws attention to the ways that tactical engagement with aspects of HCI evaluation methodology shapes and bears consequences for critically oriented research. We reflect on three of our own experiences evaluating critically oriented designs and trace challenges that we faced to the ways that sensibilities about generalizable knowledge are manifested in HCI evaluation methodology. Drawing from our own experiences, as well as other influential critically oriented design projects in HCI, we articulate some of the trade-offs involved in consciously adopting or not adopting certain normative aspects of HCI evaluation. We argue that some forms of this engagement can hamstring researchers from pursuing their intended research goals and have consequences beyond specific research projects to affect the normative discourse in the field as a whole. Author Keywords Critical technical practice; critically oriented HCI, evaluation INTRODUCTION In 1987, two graduate students at MIT published a paper describing a new way to construct intelligent programs in the major annual Artificial Intelligence (AI) conference [2]. This paper became widely influential, spawning a new area of AI research and nine years later named the most important paper in the conference that year. It was not, however, influential in the way that at least one of the authors intended. For Philip Agre, the paper represented not only a breakthrough in the technical development of AI, but also a critical reframing of AI as a research program. As later worked out in Computation and Human Experience, for Agre the primary goal of this work had been to critically analyze the limitations of the philosophical conceptualizations undergirding technical work in AI and to offer a new conceptual alternative, embodied in technical language that engineers could understand. This approach Agre termed critical technical practice. Yet, to his frustration, while AI enthusiastically picked up on the technical work, the fundamental critical reconceptualization of human activity embodied in the technology was not picked up; eventually the work was reabsorbed into what Agre saw as business as usual in AI [44]. While critical technical practice as a concept did not have the lasting influence on AI that Agre had hoped, in recent years the idea of deeply coupling technical design work with critical reflection on its conceptualization has been picked up in a neighboring field, Human-Computer Interaction (HCI). For example, within HCI, Dourish cites Agre's notion of critical technical practice as a major inspiration for his landmark work Where the Action Is [16, p 4]. Sengers et al.'s articulation of reflective design [41] expands critical technical practice to be used not only to work toward the resolution of specific technical impasses, but as a core method throughout all phases of technology’s design and use. The engagement of critical practice within technology design specifically and HCI more broadly has also been spurred through research drawing on other critical traditions. For example, Bardzell, Bardzell, and Blythe integrate criticism and critical theory into design [5,6,7,10]; DiSalvo and Hirsch draw on critical and political traditions in the arts to frame new opportunities and roles for design [15,23]; and Pierce and Löwgren use philosophy of technology and design theory to articulate and execute critical approaches to interaction design [35,36,30]. In short, there is a foment of critical approaches in HCI which aim to push forward HCI research while simultaneously troubling some of its foundational assumptions. Thus, in contrast to Agre's impact within AI, in HCI critically oriented research has clearly achieved lift-off as a vibrant research program. Paste the appropriate copyright/license statement here. ACM now supports three different publication options: • ACM copyright: ACM holds the copyright on the work. This is the historical approach. • License: The author(s) retain copyright, but ACM receives an exclusive publication license. • Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM. This text field is large enough to hold the appropriate release statement assuming it is single-spaced in TimesNewRoman 8 point font. Please do not change or modify the size of this text box. Every submission will be assigned their own unique DOI string to be